1. Technical Field
The present invention is generally directed to an improved data processing system. More specifically, one aspect of the present invention is directed to a system and method for optimizing a storage system, such as a file system, to support short data lifetimes, e.g., short file lifetimes or short object lifetimes. A second aspect of the present invention is directed to a system and method for optimizing a storage system, such as a file system, using priority based retention of data objects, e.g., files, so as to support full utilization of storage space.
2. Description of Related Art
Early file systems were designed with the expectation that data would typically be read from disk many times before being deleted. Therefore, on-disk data structures were optimized for reading of data. However, as main memory sizes increased, more read requests could be satisfied from data cached in memory. This motivated file system designs that optimized write performance rather than read performance. However, the performance of such system tends to suffer from overhead due to the need to garbage collect current, i.e. “live,” data while making room for areas where new data can be written.
New types of systems are evolving in which, in addition to reading and writing of data, creation and deletion of data are important factors in the performance of the system. These systems tend to be systems in which data is quickly created, used and discarded. These systems also tend to be systems in which the available storage system resources are generally fully utilized. In such systems, the creation of data and deletion of this data is an important factor in the overall performance of the system.
However, known file systems, which are optimized for data reads or, alternatively, data writes, do not provide an adequate performance optimization for this new breed of systems. Therefore, it would be advantageous to have a system and method that optimizes, in addition to data reads and writes, the creation and deletion of data.
All file systems have the capability for the explicit deletion of files by a program or user. Some file systems have provision for a timed delete of a file, previously scheduled by a user or program. If more files are created than deleted, eventually the system will fill, and writing new files is no longer possible. The current state of the art is tools that an administrator can use to explicitly delete files. The implication is that an administrator is forced to make decisions about the value of objects, and instigate deletion of lower value files. Therefore, it would be advantageous to have a system and method that automatically selects data to delete, retaining the most highly valued data that can fit into a file system at any given time.